Parameter uncertainty

Shen Cheng

2026-02-02

What is parameter uncertainty?

What is parameter uncertainty?

  • How confident you are in your parameter estimates \((\theta, \Omega, \Sigma)\)?
  • Typical statistics:
    • Standard errors (SE).
    • Relative standard errors (RSE).
    • Confidence intervals (CI).
  • Parameter estimates \((\theta, \Omega, \Sigma)\) are obtained with uncertainty
    • Parametric: Variance-covariance matrix (e.g., NONMEM .cov file).
    • Non-parametric:
      • Log-likelihood profiling (LLP)
      • Non-parametric bootstrap
      • Sampling-importance resampling (SIR)
      • Bayesian posterior distribution

Variability vs. uncertainty

  • Both were commonly represented by probability distribution.

  • Differ conceptually1:

    • Variability:

      • Inherent difference in the system.
      • Often cannot be reduced with more data.
      • Typically referring to \(\Omega\) and \(\Sigma\) estimates.
    • Uncertainty:

      • How confident we are in our estimates of the system.
      • Often can be reduced with more data
      • Typically refers to SE, RSE or CI of each parameter, including \(\theta\), \(\Omega\) and \(\Sigma\).

Variability vs. uncertainty

Overview of methods

Methods to assess parameter uncertainty

G cluster_0 $EST METHOD=FOCE/SAEM/IMP cluster_1 $EST METHOD=BAYES/NUTS Distributional Assumption Distributional Assumption $COV $COV Distributional Assumption->$COV No Distributional Assumption No Distributional Assumption Non-Parametric Bootstrap (BS) Non-Parametric Bootstrap (BS) No Distributional Assumption->Non-Parametric Bootstrap (BS) Log-Likelihood Profiling (LLP) Log-Likelihood Profiling (LLP) No Distributional Assumption->Log-Likelihood Profiling (LLP) Sampling Importance Resampling (SIR) Sampling Importance Resampling (SIR) No Distributional Assumption->Sampling Importance Resampling (SIR) Posterior Distribution Posterior Distribution Parameter Uncertainty Parameter Uncertainty Parameter Uncertainty->Distributional Assumption Parameter Uncertainty->No Distributional Assumption Parameter Uncertainty->Posterior Distribution

Methods to assess parameter uncertainty

  • METHOD=FOCE/SAEM/IMP
    • Distributional assumption
      • $COV: assymptotic covariance matrix from covariance step.
      • Good approximation, may not be obtained even if the model is good ($COV step failed)1.
      • Less realistic: always symmetrical (est \(\pm\) 1.96*se).
    • No distributional assumptions: Assymmetric, more realistic.
  • METHOD=BAYES/NUTS:
    • Directly acquire posterior distribution with no distributional assumption.

Methods to assess parameter uncertainty

G cluster_1 $EST METHOD=BAYES/NUTS cluster_0 $EST METHOD=FOCE/SAEM/IMP Distributional Assumption Distributional Assumption $COV $COV Distributional Assumption->$COV No Distributional Assumption No Distributional Assumption Non-Parametric Bootstrap (BS) Non-Parametric Bootstrap (BS) No Distributional Assumption->Non-Parametric Bootstrap (BS) Log-Likelihood Profiling (LLP) Log-Likelihood Profiling (LLP) No Distributional Assumption->Log-Likelihood Profiling (LLP) Sampling Importance Resampling (SIR) Sampling Importance Resampling (SIR) No Distributional Assumption->Sampling Importance Resampling (SIR) Posterior Distribution Posterior Distribution Parameter Uncertainty Parameter Uncertainty Parameter Uncertainty->Distributional Assumption Parameter Uncertainty->No Distributional Assumption Parameter Uncertainty->Posterior Distribution

Log-likelihood profiling (LLP)

Log-likelihood profiling (LLP)1

  • Assumption: changes in OFVs (dOFVs) with different parameter values follow a chi-square distribution.

  • To obtain CI of a parameter (e.g., \(\theta_1\)):

    • Fix value of \(\theta_1\) other than its final estimate while keep all other parameter unfixed.

    • Refit the model to obtain the OFV.

    • Find values of \(\theta_1\) that changes OFV by, for example, 3.84 (P value=0.05 with df=1 assuming a chi-square distribution).

LLP advantages and disadvantages

  • Advantages:
    • Do not assume symmetric normal distribution
    • Method available in open source platform: PsN
  • Disadvantages:
    • Assume chi-square distribution for dOFV, which may only be approximately true for NLME1.
    • Rarely used in PMx modeling
    • One parameter at a time

LLP implementations in PsN

After NONMEM modeling fitting, LLP can be executed using PsN with command like:

llp mod1.ctl -thetas=1,3 -rplots=2 -clean=0 -min_retries=3
  • This will evaluate uncertainty of THETA1 and THETA2 using LLP method with first guess of SE using mod1.lst.
  • -rplots=2 generates a few basic plots for output.
  • -clean=0 keeps all NONMEM fitting records.
  • -min_retries=3 usually needed to make sure each model fitting is successful.
    • If NONMEM fails, rerun the same sub-problem, up to 3 times, with reset initial conditions.”

Non-parametric bootstrap

Non-parametric bootstrap

  • Assumption: random difference in observations that leads to uncertainty in parameter estimates.

  • To obtain CI

    • Imagine an original dataset ({A,B,C,D,E}) with 5 subjects (i.e., ID in NONMEM).
    • Resample and refit each dataset:
      • Bootstrap Sample 1: ({A,C,C,D,E}): new set of parameter estimates 1
      • Bootstrap Sample 2: ({B,B,D,E,A}): new set of parameter estimates 2
      • Bootstrap Sample 3: ({C,D,E,E,E}): new set of parameter estimates 3
      • Bootstrap Sample 1000: ({B,D,A,E,E}): new set of parameter estimates 1000
    • Summarize 2.5% and 97.5% percentiles across all new sets of parameter estimates

Non-parametric bootstrap advantages and disadvantages

  • Advantages:
    • Do not assume symmetric normal distribution.
    • Obtain uncertainty for all parameters at once.
    • Method available in open source platforms:
  • Disadvantages:
    • Computationally expansive
    • Not appropriate to use when:
      • Small subject number (N < 10)
      • Frequentist prior are used
      • Performing model-based meta-analysis (MBMA)
    • Need to be cautious about the covariate stratification
    • Complex model: whether to include a parameter vector when estimation failed1

Non-parametric bootstrap implementations in PsN

After NONMEM modeling fitting, non-parametric bootstrap can be executed using PsN with command like:

bootstrap mod1.ctl -samples=500 -seed=12135 -rplots=2
  • This will evaluate uncertainty of all model parameters based on 500 refits of the NONMEM model.
  • -rplots=2 generates a few basic plots for output.

Sampling importance resampling (SIR)

Sampling importance resampling (SIR)1

Importance ratio (IR):

\[IR = \frac{e^{-0.5*dOFV}}{relPDF}\] \[dOFV = OFV_{i} - OFV_{final}\] \[relPDF = \frac{PDF_{i}}{PDF_{final}}\]

  • Rule of thumb:

    • m is large enough (500-1000): similar to 500-1000 refit in bootstrap.

    • M/m ratio is important for SIR validity (5:1 is a reasonable starting point).

Where to find a SIR Proposal distribution?

  • NONMEM covariance step ($COV)-Most typical.
  • Limited number (e.g., N=10) of non-parametric bootstrap.
  • Educated guess.
  • LLP1.

SIR diagnostics1

  • Assumption: If the parameter vectors were representative of the true uncertainty, their dOFV distribution should follow a Chi-square distribution with a certain degree of freedom.
  • Inverse cumulative density function (CDF) plot of a Chi-square distribution.
    • If the associated degree of freedom (DF) is smaller, the corresponding line is lower on the graph.

SIR diagnostics1

  • The \(DF_{reference} = N_{parameters}\) in a model, an acceptable SIR should have dOFV plot with:
    • \(DF_{proposal}>DF_{reference}\): blue dash line higher than gray solid line.
      • If not, increase variance of proposal distribution.
    • \(DF_{final} \leq DF_{reference}\): blue solid line lower than gray solid line.
      • If not, increase M/m ratio.

SIR advantages

  • Do not assume symmetric normal distribution.
  • Obtain uncertainty for all parameters at once.
  • Overcome non-parametric bootstrap drawbacks:
    • Faster.
    • No model refit: do not need to consider convergence.
    • Do not need to consider covariate imbalance and small sample size.
    • OK to use in MBMA or in the presence of frequentist prior.
  • Well-developed diagnostics.
  • Method available in open source platforms:

SIR disadvantages

  • Still slow, although faster than bootstrap.
  • Sensitive to proposal distributions1.
  • May need multiple attempts based on diagnostics.

SIR implementations in PsN

After NONMEM modeling fitting, SIR can be executed using PsN with command like:

sir -samples=2500 -resamples=500 -rplots=2 mod1.ctl
  • This will evaluate uncertainty of all model parameters based on 500 resampled parameter sets with a M/m ratio of 5:1 (2500:500).

  • -rplots=2 generates dOFV and a few other diagnostics.

  • What if the diagnostics is not satisfied?

    • If \(DF_{proposal}<DF_{reference}\), inflate proposal distribution variance.

      • sir -samples=2500 -resamples=500 -theta_inflation=1.5 -omega_inflation=1.5 -sigma_inflation=1.5 -rplots=2 mod1.ctl
    • If \(DF_{final}>DF_{reference}\), increase M/m ratio.

      • sir -samples=5000 -resamples=500 -rplots=2 mod1.ctl

SIR Practical Suggestions

  • Start with a small batch of sampling / resampling (e.g., 500/100) and check the diagnostics.
    • optimize the SIR conditions (e.g., proposal distribution variance, sampling to resampling ratio) as needed.
  • Perform final SIR using the optimized SIR conditions.

Hands-on SIR

Multiple-iteration SIR

Multiple-iteration SIR1

Multiple-iteration SIR implementations in PsN

After NONMEM modeling fitting, multiple iteration SIR can be executed using PsN with command like:

sir -samples=2500,2500,2500,1000,1000 -resamples=500,500,500,500,500 -rplots=2 mod1.ctl
  • This will evaluate uncertainty of all model parameters based on 500 resampled parameter sets with multiple iteration sampling and resampling with M/m ratio of 5:1 (2500:500), 5:1 (2500:500), 5:1 (2500:500), 2:1 (1000:500), and 2:1 (1000:500).
  • -rplots=2 generates dOFV and a few other diagnostics.